PolyLU: A simple and robust polynomial-based linear unit activation function for deep learning

The activation function has a critical influence on whether convolutional neural network in deep learning can converge or not; proper not only makes the faster but also reduce complexity of architecture and gets same better performance. Many functions have been proposed; however, various advantages, defects, applicable architectures. A new called Polynomial Linear Unit (PolyLU) is proposed this paper to improve some shortcomings existing functions. PolyLU meets following basic properties: continuously differentiable, approximate identity near origin, unbounded for positive inputs, bounded negative smooth, monotonic, zero-centered. There polynomial term inputs no exponential terms that reduces computational network. Compared those common like Sigmoid, Tanh, ReLU, LeakyReLU, ELU, Mish, Swish, experiments show improved accuracy over MNIST, Kaggle Cats Dogs, CIFAR-10 CIFAR-100 datasets. Test by dataset with batch normalization, improves 0.62%, 2.82%, 2.44%, 1.33%, 2.08%, 4.26% than Leaky Tanh respectively. without 1.24%, 4.39%, 2.12%, 5.43%, 15.51%, 8.10%